Development, Implementation And Testing Of A Discourse Model For Newspaper Texts

Texts of a particular type evidence a discernible, predictable schema. These schemata can be delineated, and as such provide models of their respective text-types which are of use in automatically structuring texts. We have developed a Text Structurer module which recognizes text-level structure for use within a larger information retrieval system to delineate the discourse-level organization of each document's contents. This allows those document components which are more likely to contain the type of information suggested by the user's query to be selected for higher weighting. We chose newspaper text as the first text type to implement. Several iterations of manually coding a randomly chosen sample of newspaper articles enabled us to develop a newspaper text model. This process suggested that our intellectual decomposing of texts relied on six types of linguistic information, which were incorporated into the Text Structurer module. Evaluation of the results of the module led to a revision of the underlying text model and of the Text Structurer itself. 1. D I S C O U R S E L E V E L T E X T M O D E L S A discourse-level model of a text type can be likened to an interpretation model [Breuker & Wielinga, 1986] in that it specifies the necessary classes of knowledge to be identified in order to develop the skeletal conceptual structure for a class of entities. The establishment of text-type models derives from research in discourse linguistics which has shown that writers who repeatedly produce texts of a particular type are influenced by the schema of that texttype and, when writing, consider not only the specific content they wish to convey but also what the usual structure is for that type of text based on the purpose it is intended to serve [Jones, 1983]. As a result, one basic tenet of discourse linguistics is that texts of a particular type evidence the schema that exists in the minds of those who produce the texts. These schemata can be delineated, and as such provide models of their respective text-types which we suggest would be of use in automatically structuring texts. The existence of and need for such predictable structures in texts is consistent with findings in cognitive psychology suggesting that human cognitive processes are facilitated by the ability to 'chunk' the vast amounts of information encountered in daily life into larger units of organized data [Rumelhart, 1977]. Schema theories posit that during chunking we recode individual units of perception into increasingly larger units, until we reach the level of a schema. Humans are thought to possess schema for a wide range of concepts, events, and situations [Rumelhart, 1980]. Discourse linguists have extended this notion to suggest that schema exist for text-types that participate regularly in the shared communication of a particular community of users. What is delineated when a text schema is explicated is its discernible, predictable structure, referred to as the text's Superstructure. Superstructure is the text-level syntactic organization of semantic content; the global schematic structure; the recognizable template that is filled with different meaning in each particular example of that texttype [van Dijk, 1980]. Among the text-types for which schemas or models have been developed with varying degrees of detail are: folk-tales [Propp, 1958], newspaper articles [van Dijk, 1980], arguments [Cohen, 1987], historical journal articles [Tibbo, 1989], and editorials [Alvarado, 1990], empirical abstracts [Liddy, 1991], and theoretical abstracts [Francis & Liddy, 1991].